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Evocative evaluation of assessment data is essential to quantify the achievements at course 
and program levels. The objective of this paper is to design performance metrics and 
respective formulas to quantitatively evaluate the achievement of set objectives and ex¬ 
pected outcomes at the course levels for program accreditation. Even though assessment 
processes for accreditation are well documented but existence of an evaluation process is 
assumed. This work provides performance metrics such as attainment, student achieve¬ 
ment, and x-th percentile for the evaluation of assessment data at course and program 
level. Then, a sample course data and uniformly distributed synthetic data are used to an¬ 
alyze the results from designed metrics. The primary findings of this work are twofold: 
(i) analysis with sample course assessment data reveals that qualitative mapping between 
marks obtained in assessments to the defined outcomes is essential for meaningful re¬ 
sults, (ii) analysis with synthetic data shows that higher values of one metric does not 
imply higher values of the other metrics and they depend upon the obtained marks dis¬ 
tribution. In particular, for uniformly distributed marks, achievement < attainment for 
meanO fUnif ormDistr. < averageMarks < passingThreshold(a). Authors hope 
that the articulated description of evaluation formulas will help convergence to high quality 
standard in evaluation process. 
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1. INTRODUCTION 

Any educational program starts with a mission statement, objectives, and the program or student outcomes. 
Mission statement describes the broad goals of the program. Program educational objectives (PEO) are statements 
regarding the expected positions that students may attain within a few years of graduation, whereas Student outcomes 
(SO) are the expected skills at the time of graduation. SOs are directly tied with the course learning outcomes (CLO) 
which are the expected skills of a student at the end of the course in a program; more formal definitions can be found 
at [1], 

Assessment and evaluation are integral parts of the quality assurance, continuous improvement, and the 
accreditation. Assessment is defined as one or more processes that identify, collect, and prepare the data necessary for 
evaluation. Evaluation is defined as one or more processes for interpreting the data acquired through the assessment 
processes in order to determine how well SOs are being attained [1], 

Many authors have published their work on continuous improvement, data collection and assessment strate¬ 
gies but there is no work in the literature that focuses on the evaluation of the assessment data at course and program 
levels. A complete procedure for ABET accreditation for Engineering programs at Qassim University has been pre¬ 
sented in [2]. It gives the detailed implementation of the continuous improvement process, effecting major changes in 
the educational plan, curricular content, facilities, activities, teaching methodologies, and assessment practices. But 
this paper does not go into the details of evaluation process. Olds et al. [3] examine many possible assessment methods 
in comply with ABET criteria. They categorize assessment methodologies into descriptive and experimental studies. 
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and provide various effective types of assessments for engineering education without the insights of evaluation strate¬ 
gies. An overview of program level assessment for continuous improvement is given in [4], It explains assessment 
process in five steps: from the identification of educational objectives to the measurement of assessment data but none 
of these step provides the evaluation methodology. A complete assessment process from writing good learning out¬ 
comes, mapping between course level outcomes and program level outcomes, and data collection using various direct 
and indirect assessment methods is illustrated in [5] and [6], In [7] authors present an assessment plan and continuous 
improvement process at James Madison University. They have introduced course assessment and continuous improve¬ 
ment (CACI) reports at course-level and student outcomes summary report (SOSR) at program-level. This paper shows 
some sample reports and assessment templates but does not discuss the evaluation process. A web-based tool has been 
introduced in [8] for outcome-based open-ended and recursive hierarchical quantitative assessment. This quantitative 
assessment is used to structure outcomes and measures into a leveled hierarchy, with course outcomes at the bottom 
and more general objectives at the top. A general curriculum outcome (GCO) layer has been added between course’s 
outcomes and program or student’s outcomes. In [9] both direct and indirect measures are used to collect and analyze 
data to assess the attainments of the student outcomes. To ensure data integrity, a set of rubrics with benchmarks and 
performance indicators at both the program and curriculum levels are developed. Each outcome has been assessed for 
different levels (introductory, beginning, developing, proficient, exemplary) and from different sources. The article 

[10] presents discussions on writing learning outcomes and to assess soft skills in engineering education. The paper 

[11] describes the assessment techniques and the mapping of CLO to SO without the insight of evaluation process. A 
case study [12] describes the features that contribute to assessment quality at the programme, course and task level. 
This case study has a particular focus on the technical such as task analysis and task relationship patterns. Another 
case study [13] presents a health science program reform and evaluation. It discusses potential for evaluation to es¬ 
tablish responsive communication between students, teaching staff and programme administrators, ensuring a match 
between the intended, implemented and attained curriculum. A web-based Instrumental Module Development System 
(IMODS) for outcome-based course design has been presented in [14]. It defines the learning outcomes, mapping, 
and assessment process but the evaluation of assessment data is not explained. Ibrahim et al. [15] work is close to 
our proposed design. It consists of a web-based tool to measure the mean, standard deviation, and the achievement 
through course assessments’ data. They formulate these evaluation metrics by assuming normally distributed dataset 
only. These formulas cannot be applied to other distributions that may occur in real practice. None of these works 
however, provide the detail of general evaluation metrics and their use in course and program assessment. 

The rest of the article is organized as follows: the next section describes the performance metrics formulation, 
course level evaluation based on performance metrics are explained in section III, analysis and interpretations are 
discussed in section IV and conclusions are drawn in section V. 


2. PERFORMANCE METRICS FORMULATION 

Let Ai j m be the marks obtained by student m in question j of assessment i (homework, assignment, quiz, 
midterm, or final etc). Here i can take the values, i = 1,2,..., /, j can take the values j, = 1, 2,...,./,, and m can take 
the values m = 1, 2,..., M. I ,./,. M are the total number of assessments, questions in assessment i, and the students, 
respectively. 

For quantitative analysis, question is a basic unit of computation for assessment. Average score of question j 
in assessment i that has M students is given by 


1 


M 

53 Aij tTn 

m— 1 


Bi j can be written in vector form as 


B = Bi Bo 


B, 


I lxL 


0 ) 


( 2 ) 


Passing threshold (PT) could be absolute, relative or composite [8], such that the PT of question j in assessment i is 
given by one of the following: 


PT tJ = a x Q*j 

PTi.j — Bij 

PTi tj = min {B itj ,a x } 


(3) 

(4) 

(5) 


IJEREVol. 5, No. 3, September 2016: 235-245 





IJERE 


ISSN: 2252-8822 


□ 


237 


where Qj 0 J is the maximum or total marks of question j in assessment i and 0 < a < 1. The maximum, minimum, 
standard deviation, and x-th percentile of question j in assessment i are calculated as 


A 


i,j,max 


A. 


A ■ ■ 

.per 


max A i ,. 

m ’ J ’’ 

min A , , • 

m ’ J 

stdevAij tm 

percentile(Aij, : ,x) 


Course learning outcomes describe what students are expected to learn in a course. A mapping between CLO 
and assessment questions is required to compute the attainment of the course CLOs. If a course covers N number of 
CLOs then n th CLO is written as CLO,, , n = 1,2, ...N. The three dimensional matrix A is converted into a two 
dimension matrix A as 

A = [ Af A^ . . Aj ] (6) 

where Af is the transpose matrix of i th assessment matrix A j x m- Matrix A has the dimension M x L, where M is 
the total number of students and L = ^A =1 Ji is a total number of questions in all assessments. 

CLO to SO mapping matrix is by 


cs ha 

cs 1>b ■ ■ ■ 

l 

to 

o 



cs 2 , a 

cs 2 , b ••• 

CS N - hk 


(7) 

CS N ,a 

• • • cs NJ 

cs N , k 



CS n , a > 

0 if CLO n maps 

to SO a, otherwise CS n>a 

= 0. A non-zero 


CS = 


value is a relevance of the CLO to compute the SO. It can take values 1,2,3, for low, moderate, and high relevance, 
respectively. 

Similarly, the CLO to question mapping is given by the following matrix 


CQ 


CQ M 

CQi , 2 

CQi, l 

CQ 2 p 

cq 2 , 2 

CQn-i,l 

CQn,i 

CQn,l-i 

CQn,l 


( 8 ) 


Rows of above matrix contain binary variables CQ n j that represent the mapping of n th CLO with question l, where 
l maps to j th question of an assessment i. CQ n j = 1 if CLO n maps to question l. 


Student marks in assessment questions are used to compute how well the students have done and what per¬ 
centage of students have met a certain criteria. Every question contributes to one or more CLOs and every CLO 
contributes to one or more student outcomes (SO) as shown in (8) and (7) respectively. 


2.1. CLO Attainment 

This metric is about how well the students have done, in percentage, for each CLO. Attainment of a CLO is 
derived from the average marks obtained divided by total marks for all questions that maps to the CLO. 

Let Ql°* be the maximum or total marks of question j in assessment i. In general, for assessment i, 

Q l ot = [ Q£? Q$ ■ • Q Tj ] (9) 

and 

Q tot = [ Qi ot Ql ot • • Q/ of ] lxi (10) 

Then, the percentage of CLO attainment for n th CLO is given by 

attainment CLO n [%] = x ^qq 

Ef=i Ql ot * CQm 

The operator * is used for element-wise multiplication. 
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2.2. CLO Weightage Information 

In order to get meaningful results, one should design the CLOs such that there is a uniform distribution of 
the marks over CLOs in questions to CLO mapping. For example, if a course contains four CLOs then, ideally, each 
CLO should get 25% weightage. 

The ideal case of uniform distribution of marks over the CLOs is seldom realized. In these situations, the 
CLO weightage information renders a fair picture of % CLO attainment. The percentage weightage of n th CLO is 
given by 

„ w(CQ nJ )Ql°f 

WeightageCLO n = ^ - — x w(Ai) (12) 

i 1 

where w(CQ n j) is the weight of CLO n in question j, w(Ai) is the weight of assessment i, and A \ ot is the total 
marks of assessment i. 


2.3. Student Achievement per CLO 

Student Achievement per CLO is defined as the percentage of students who are above the expected level as 
shown in (5). Expectation or target is a design parameter, one choice of the target could be min (Bij, 0.7 x i.e., 

the minimum of the average obtained marks and the 70% of the total marks [8], 

It counts the number of students who met the criteria by comparing each student marks in a question j of 
an assessment i. If the marks obtained Aij tTn are greater than the passing threshold PTp, then it increments the 
counting variable CPS (count pass student) by 1. Finally, CPSij or CP Si 1 contains number of passed students for 
each question j in assessment i. Therefore, the average student achievement per CLO is given by 


SA_CLO n 


Eli A EjU CPSjj X CQ U}i j x 
Si=l Zj =1 Qi°j X CQn,i,j 


where M,; is the number of students that participated in the assessment i. 


(13) 


2.4. Student Perception of CLO Attainment 


A course survey is conducted at the end of each semester to gauge students’ perception of how well the 
CLOs were covered in the course. It is the average of CLO perception from the students. For each CLO, students 
provide their input on the scale of 1 — 5 where 1 means CLO is not achieved and 5 means CLO is achieved completely. 
Summary of responses is given in following matrix. 


- 1 

Co 

p 

SC 1,2 

sc hN ■ 

SC 2 ,i 

SC 2 , 2 

SCm-i,n 

_SC M ,i 

SCm,n-i 

SCm,n 


Student’s perception of n th CLO attainment is given by 

1 M 

SEJJLO n = — ^ 


(14) 


(15) 


2.5. x-tli Percentile Marks per CLO 

x-th Percentile Marks per CLO is defined as the weighted average of x — th percentile marks divided by total 
marks of the questions that map to particular CLO. 

Let xPij be the x-th percentile marks of question j in assessment i. In general, for assessment i we have 


xPj = [ xP u xP i 2 


and 

xP = [ xPi xP 2 

[ CPSi is a row vector form of CPSij, similar to (6) or (2) 


xP,:,j 


xP / 


1 xL 


06) 

(17) 
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Table 1. Basic Variables to compute evaluations metrics for a sample course. CLOs and questions mapping (8) is 
shown in first two rows 




midterm 


Quizl 

Quiz4 

Quiz3 



HW1 



Quiz2 



Final 



Class participation 

CLOs Covered 

1,2,3 

1 

2 

3,4 

1 

6 

5 

2 

2 

3 

3 

3 

2 

5 

5 

5 

6 

6 

1-6 

Question No. 

1 

2 

3 

4 

1 

1 

1 

1 

2 

3 

4 

5 

1 

1 

2 

3 

4 

5 

1 

Question Marks 

8 

8 

8 

8 

4 

4 

4 

0.4 

0.4 

0.4 

0.4 

0.4 

4 

5 

10 

10 

10 

10 

5 

Actual Average 

5.22 

6.6 

6.99 

6.69 

3.25 

3.664 

3.12 

0.4 

0.4 

0.4 

0.352 

0.128 

3.68 

1.9 

8.3 

7.2 

6.1 

9.3 

4.7 

Passing Threshold (PT) 

5.22 

5.6 

5.6 

5.6 

2.8 

2.8 

2.8 

0.28 

0.28 

0.28 

0.28 

0.128 

2.8 

1.9 

7 

7 

6.1 

7 

3.5 

No. of Students Above PT 

6 

9 

10 

9 

9 

10 

6 

10 

10 

10 

8 

4 

9 

7 

6 

4 

4 

10 

10 

Minimum Marks 

1.2 

4.2 

6 

4.5 

1.3 

3.04 

2.4 

0.4 

0.4 

0.4 

0.16 

0 

2.4 

0 

5 

5 

1 

8 

4.27 

Maximum Marks 

7.35 

7.5 

7.5 

7.5 

4 

3.84 

3.84 

0.4 

0.4 

0.4 

0.4 

0.32 

4 

4 

10 

10 

10 

10 

4.9 

Standard Deviation 

1.78 

0.95 

0.62 

0.92 

0.78 

0.25 

0.47 

0.00 

0.00 

0.00 

0.10 

0.12 

0.44 

1.37 

1.95 

1.60 

2.91 

0.64 

0.21 

50th Percentile Marks 

5.85 

6.75 

7.35 

6.9 

3.3 

3.76 

3.2 

0.4 

0.4 

0.4 

0.4 

0.08 

3.84 

2 

9.5 

6.5 

5.5 

9 

4.79 


> xP > 

xPercentileC LO n = —' _1 - 

X'L n tot 
2^i=i v; 


(18) 


2.6. SO Attainment 

By using CLO-SO mapping in (7), course level SO assessment can be achieved. SO attainment for an SO is 
computed from the CLO attainment of all CLOs that map to the SO. SO attainment is defined as the weighted average 
of CLOs attainment (in %). 

Y attainmentCLOiWi 

attainmentSO n = -=-— (19) 

E,: eC " u, i 

where C n is the set of CLOs that map to SO n and w, is the weight (or relevance) of i th mapping between CLO and 


SO. 

2.7. Student Achievement per SO 

It is defined as weighted average of student achievement of CLOs (in %) that map to a particular SO. 


SA_SO n 


Eiec" SA.CLOiWi 


( 20 ) 


2.8. Student Perception of SOs Attainment 

Student perception of SOs attainment gives an indirect measurement of SO attainment. This metric is derived 
from student perception of CLOs attainment in (15) and the CLO-SO mapping in (7). 


SE-SO n 


£ ieC n SEJJLOiWi 

£ ie c» w i 


( 21 ) 


2.9. x-th Percentile per SO 

x-th Percentile per SO uses CLO-Average x-th Percentile % marks with CLO-SO mapping in (7). 


xPercentileSOn = 


E 


ieC’ 


. xPercentileCLOjWj 


E 


( 22 ) 


ieC n 


3. COURSE LEVEL PERFORMANCE EVALUATION 

Direct assessment of an academic program is performed by evaluation of courses in the study plan. If not 
all, at least a selected subset of the courses is required to find out program’s success level. Previous section presented 
formal formulations of the performance metrics that can be used in course evaluation. This section discusses an 
implementation of these metrics in evaluation of a sample course. The section starts with setup required for evaluation 
followed by evaluation results and concludes by discussing issues and concerns. 
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Table 2. Mapping of Course Learning Outcomes (CLOs) to Student Outcomes (SOs) (7) as Course Assessment Matrix 


CLO 

a 

b 

c 

d 

e 

f 

g 

h 

i 

j 

k 

1 



2 









2 

2 

1 






1 




3 



3 









4 





2 




1 



5 

3 

2 









2 

6 

3 

3 

2 










Table 3. Computation of SO attainment from CLO attainment using table 2 CLO-SO mapping 


CLO 

a 

b 

c 

d 

e 

f 

g 

h 

i 

j 

k 

1 



79.08 









2 

82.91 

82.91 






82.91 




3 



78.78 









4 





87.62 




87.62 



5 

74.18 

74.18 









74.18 

6 

81.94 

81.94 

81.94 









SO attainment 

79.68 

79.68 

79.94 


87.62 



82.91 

87.62 


74.18 

Weighted SO attainment 

79.27 

79.52 

79.77 


87.62 



82.91 

87.62 


74.18 

Relevance 

3 

2 

3 


2 



1 

1 


2 


3.1. Course Setup for Evaluation 

Course evaluation is computation of performance metrics from basic variable of the course and perform 
analysis. To compute metrics defined in section II from collected data, each course must have well defined CLOs, 
CLO to SO mapping as in (7), CLO to question mapping in each assessment as in (8), and passing threshold as defined 
in (5). Table 1 shows basic variables of a sample course. First two rows show mapping between CLOs and questions 
for all assessments conducted in the course. Table 2 shows mapping between CLOs and SOs defined by the course 
designer. A numeric value in a cell represents a relationship between a CLO and an SO. A value of 1, 2, or 3 indicates 
that a CLO addresses an SO slightly, moderately, or substantively. Passing threshold is set to min(avg, 70%), which 
is used in computation of student achievement per CLO (13) and student achievement per SO (20). 

3.2. Performance Evaluation 

This section presents computed values of metrics defined in section II for the sample course. 

3.2.1. CLO Attainment 

CLO attainment for the sample single section course is shown in Figure 1. CLO attainment quantifies the 
student attainment level of particular CLO through the percentage marks allocated to that CLO. Since this is a per¬ 
centage value of average marks obtained in the questions maps to a particular CLO, therefore, it is necessary to either 
distribute the marks uniformly over the CLOs or give an explicit evidence of CLO to marks ratio. 




a 

H 

1 

1 1 

■ J 

d 

■ 



- 1 1 . 



1 

1 

n 

n 



■ CLO 

Attainment 

[%] 

■ Student 
achievement 
[%] 

■ 50th- 
percentile % 
marks 

■ Student 
perception 
of CLO [%] 


CLOl CL02 CL03 CL04 CL05 CL06 


Figure 1. CLO Performance Evaluation 
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3.2.2. CLO Weightage 

The CLO weightage for a sample single section course is assumed as CLOl 15%, CL02 16%, CL03 9%, 
CL04 5%, CL05 30%, and CL06 25%. The CLO attainment and student achievement of CLO are based on these 
weightages. 

3.2.3. Student Achievement per CLO 

Student achievement per CLO for the sample single section course is shown in Ligure 1. It is the percentage 
number of students that meet or exceed the target or expectations. There is an upper limit for the target (70%) but 
there is no lower limit and it depends upon the average marks. We can get absolute student achievement by fixing the 
target, for example, with target value of 60%. 

3.2.4. 50-th Percentile per CLO 

The 50-th percentile for the sample course is shown in Ligure 1. It shows the percentage median marks for 
each CLO. 


3.2.5. Student Perception of CLOs Attainment 

Lor each CLO, student perception of CLO attainment can be on the scale of 1 to 5, 1 mean strongly disagree 
to 5 for strongly agree. Student perception of CLO attainment for the sample course is shown in Ligure 1. There were 
10 students but 8 participated in the course survey. 


3.2.6. SO Attainment 

Bar graphs for SO attainment are shown in Ligure 2. These levels are averages of CLO attainments that map 
to particular SO, therefore, the health of CLO attainments and CLO-SO mapping is critical. 



Ligure 2. SO Performance Evaluation 


3.2.7. Student Achievement per SO 

Student achievement for the sample course is given in Ligure 2. This is a derived value from CLO achieve¬ 
ments and depicts the percentage number of students achieved the set target averaged over the CLOs mapped to that 
SO. 

3.2.8. 50-th Percentile per SO 

The 50-th percentile per SO values are derived from 50-th percentile per CLO. Ligure 2 depicts the 50-th 
percentile or median marks for each mapped SO. 

3.2.9. Student Perception of SOs Attainment 

The student perception of SOs attainment is shown in Ligure 2. It is an indirect measurement obtained from 
the course exit survey where students provide their feedback about the CLOs attainment. 
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3.3. Issues and Guidelines 

Course designer is responsible to establish quality mapping between CLOs and SOs. Course instructor is 
responsible for CLOs to questions mapping for all assessments. Quality of these mappings have direct impact on the 
evaluation results as discussed in rest of this section. 

3.3.1. Relationship of Questions, Marks distribution and CLOs 

It has been observed that questions to CLO mapping requires uniform marks distribution over the CLOs. 
The quantitative measurement of CLOs provides the baseline data for direct assessment, therefore, questions to CLOs 
mapping is critical in direct assessment. CLOs should be designed in such a way that they cover all the core topics 
(qualitative equality) and course assessments should cover all CLOs with uniform marks distribution over the CLOs 
(quantitative equality). Similar measures are required in capstone project rubrics’ design. Capstone project is an 
important entity of program in which students apply the knowledge gained during the course of the program to solve 
the engineering problems. The capstone project rubrics map to CLOs and these CLOs usually cover all the SOs. 
Since the sample size in this assessment is not as large as of direct assessment therefore results may differ in these 
assessments. 

3.3.2. Questions to CLOs Mapping Approaches 

Due to the many-to-many mapping between questions and CLOs, a common question arises about the weights 
of a question that maps to multiple CLOs. There are three possibilities: 

• One-to-many mapping with equal weights 

• One-to-many mapping with proportional weights 

• One-to-one mapping between questions and CLOs. 

In this manuscript, equal weights have been used in questions to CLOs mapping. The proportional weights add one 
more level of complexity for the faculty and hence more chances of errors. One-to-one mapping is another attractive 
solution which eliminates the weight problem because in this case one question can be mapped to one CLO at most. 
In this technique many questions can be mapped to one CLO but converse is not possible. Proportional weights and 
one-to-one schemes require a proper design of CLOs and the mapping table between questions and CLOs. 

3.3.3. CLOs to SOs Mapping within a Course 

There are three choices: 

• One CLO can be mapped to any number of SOs without weights (one-to-many mapping without weights) 

• One CLO can be mapped to any number of SOs with weights (one-to-many mapping with weights) 

• One CLO can be mapped to one SO only (one-to-one mapping) [16] 

In this manuscript, one-to-many mapping with weights has been used as shown in Table 2. One-to-many mapping 
without weights assumes equal weights across all SOs mapped to a particular CLO. A straight forward way of mapping 
is one-to-one mapping which does not require weights but again the design of CLOs is important in this case. 

4. ANALYSIS AND INTERPRETATIONS 

This section provides a detail analysis of course level evaluations based on the formulated evaluation metrics 
using synthetic data. The implementation of these metrics have revealed several new directions and interpretations. 

Course evaluation produces quantitative values of attainment, achievement, and x — th percentile metrics for 
CLOs and SOs. For a course, relative values of these metrics provide insight into what happened in the course and zero 
value for a metric indicates that topics related to the corresponding CLO or SO are either not covered in the course 
or data was not collected for evaluation. For a multi-section course, these metrics can point to lack of coordination 
among course instructors, and difference in teaching and evaluation standards. 

If a course instructor does not cover some CLOs then corresponding metrics values will be zero as shown in 
Figure 3. For the sample course, CLO 4 and 5 are not covered and since these two CLOs maps to SO ”a”, so the SO 
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Figure 3. Comparison of CLO attainment, student achievement, and 50-tli percentile 


is also not covered in the course. For multi-section courses, a zero value for any of the defined metrics in some of the 
section indicates lack of coordination among course instructors. 

In order to explain the relationship, CLO attainment, student achievement, and 50-th have been plotted against 
the students’ average marks in Fig. 3. These graphs show the three evaluation metrics’ values for a range of average 
marks associated with a particular CLO. In this figure, the average marks are obtained from normal distribution mean 
(average) with standard deviation 5. Number of students is 30 and the results are averaged over 1000 iterations. 
From this figure, it can be seen that CLO attainment is a linear function of questions’ average marks mapped to that 
CLO. The attainment is equal to the 50-th percentile because the mean and median of normal distribution are equal. 
The composite student achievement remains almost constant up to 70% average marks due to the passing threshold 
min(averageMarks, 70%TotalMarks). For normally distributed marks, there are always 50% students below the 
average value and 50% students are above the average value, hence, student achievement remains constant at 50% 
value. When the average marks go above 70%, the passing threshold shifts from average value to 70% and all the 
students with marks greater than 70% contribute to the student achievement. At about 80% average marks, the student 
achievement reaches to 100% value because all the students even with the standard deviation of 5 now lie above 70% 
threshold. The relative achievement (passing threshold=average marks) always remains around 50%. The absolute 
achievement (passing threhold=0.7Total marks) for a = 0.7 crosses the 50% value at average marks equal to 70. 

The designed evaluation metrics give comprehensive results when considered collectively. 

1. Attainment and Achievement: The relationship between attainment and student achievement (absolute, com¬ 
posite) is linear for average marks greater than set target, i.e., for sufficiently large population size and normal 
distribution of obtained marks, high values of attainment corresponds to high values of achievements and the 
low attainment expects low achievement. Attainment and achievement are independent for achievement level 
below the threshold. If the distribution is not normal then linearity is not guaranteed. For example, if there 
are 10 students and 9 students secure 50 marks (out of 100) and one student get 10 marks then the composite 
achievement is 90% but the attainment is 46% 

2. Attainment and Percentile: The 50-th percentile (or median) gives an additional information about the health 
of attainment. It is also called location parameter. Median close to attainment indicates normal distribution of 
marks. 

3. Achievement and Percentile: If 50-th percentile (median) is equal to the target value of achievement then the 
achievement is equal to 50%. Median values above the the achievement target shows that more students have 
met the expectation and value of achievement will be high. Median values less than the achievement target 
results in the achievement level less than 50%. 

4. Attainment, Achievement, and Percentile: Attainment and 50-th percentile (median) have the same units, i.e., 
average and median marks in a particular CLO, whereas, student achievement gives the number of students. If 
attainment and median are similar (normal distribution) and have high values, then, the absolute and composite 
achievements will also be high because more number of students would have marks greater than the set target, 
whereas, the relative achievement will remain flat at 50% because of normally distributed marks. Conversely, if 
attainment and median have low levels, then, the composite achievement becomes 50%. Note that the absolute 
achievement is proportional to the attainment and median near the target and becomes independent for the 
average marks sufficiently less or greater than the target value. 
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5. CONCLUSION AND FUTURE WORK 

This work qualitatively evaluates the course assessment data using designed performance metrics (attainment, 
student achievement, x-th percentile). The main contribution of this work is to design and implement the performance 
metrics for the evaluation of assessment data. There are many published papers on the outcome-based assessment of 
course and program but none of those explicitly depicts the formulation of the evaluation metrics. 

Using the designed metrics, the first finding obtained is that meaningful results from evaluation metrics depend upon 
the qualitative mapping between marks obtained in assessments to the CLOs. CLOs definitions from the course core 
topics require qualitative equality, and the marks distribution over the CLOs requires quantitative equality. 

Analysis of the results obtained from the uniformly distributed synthetic data show that higher values of one metric 
does not imply higher values of the other metrics and they depend upon the obtained marks distribution. In particular, 
for uniformly distributed marks, achievement < attainment for meanOfUniformDistr. < averageMarks < 
passingThreshold(a). 

As future work, these performance metrics can be used to evaluate the outcome-based program assessment with the 
real data over two or three years. Some qualitative measurements (student course survey, graduate student survey, 
employer survey) can complement the study presented here. 
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